An Architecture for Multimodal Information Fusion
نویسندگان
چکیده
A. Shaikh, S. Juth, A. Medl, I. Marsic, C. Kulikowski, and J. L. Flanagan CAIP Center, Rutgers University 96 Frelinghuysen Road, Piscataway, NJ 08854-8088 [email protected] Abstract This paper presents a multimodal interface featuring fusion of multiple modalities for natural human-computer interaction. The architecture of the interface and the methods applied are described, and the results of the real-time multimodal fusion are analyzed. The research in progress concerning a mission planning scenario is discussed and other possible future directions are also presented.
منابع مشابه
Multimodal medical image fusion based on Yager’s intuitionistic fuzzy sets
The objective of image fusion for medical images is to combine multiple images obtained from various sources into a single image suitable for better diagnosis. Most of the state-of-the-art image fusing technique is based on nonfuzzy sets, and the fused image so obtained lags with complementary information. Intuitionistic fuzzy sets (IFS) are determined to be more suitable for civilian, and medi...
متن کاملFUSION AND COORDINATION FOR MULTIMODAL INTERACTION Roadmap, Architecture, Tools, Semantics
Users require more effective and efficient means of interaction with increasingly complex information and new interactive devices. This document summarizes the results of the international Dagstuhl Seminar on Coordination and Fusion in Multimodal Interaction that took place at Schloss Dagstuhl in Germany October 27 through November 2, 2001. We first outline a research roadmap in the near and lo...
متن کاملMultimodal Systems: Aspects of Events Fusion and a Taxonomy
This paper discusses multimodal systems. We have defined a taxonomy of multimodal interactions and built two example multimodal systems. We consider several levels where the fusion can take place in the software architecture of those systems. We emphasize the need for a common abstract representation to represent input data and commands, and to combine them successfully. This is still an area o...
متن کاملSemantic Based Information Fusion in a Multimodal Interface
The fusion of different input channels is a key factor for multimodal human-computer interfaces. In this work, we present a new concept for modality fusion. Our aim was an interface architecture that is restricted neither to a specific application field nor to specific input commands. We use a semantic network in combination with a dictionary for describing the possible user inputs – the intera...
متن کاملGated Multimodal Units for Information Fusion
This paper presents a novel model for multimodal learning based on gated neu-ral networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit u...
متن کامل